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Developing  a  Methodology  for  Assessing  Safety  Programs 
Targeting  Human  Error  in  Aviation 


“I  believe  that  the  past  is  prologue . In  our  recommendations  we  try  to  take  what  we  have 

learned  and  correct  situations  so  it  shouldn’t  happen  again.  ” 

— Former  National  Transportation  Safety  Board  Chairman  James  Hall  (1996) 


INTRODUCTION 

Indeed,  the  National  Transportation  Safety  Board 
(NTSB),  Federal  Aviation  Administration  (FAA),  and 
other  safety  organizations  have  committed  extraordinary 
resources  to  prevent  civilian  aviation  accidents.  As  a  result, 
aviation  in  the  U.S.,  particularly  commercial  aviation,  has 
become  one  of  the  safest  modes  of  transportation.  Still, 
accidents  can  happen,  often  repeating  the  same  sequence 
of  events  played  out  many  times  before.  As  a  result,  we 
are  often  left  with  the  regrettable  truth  that  there  are 
really  very  few  “new”  accidents,  just  different  players. 
Perhaps  that  is  why  Chairman  Hall  chose  the  operative 
term  “shouldn’t”  rather  than  “won’t”  in  1996. 

So  if  there  really  are  few  “new”  accidents,  why  has  the 
aviation  accident  rate  remained  relatively  stable  over  the 
last  several  years?  After  all,  if  we  already  know  what  the 
problem  is,  why  have  we  been  unable  to  fix  it?  Perhaps 
it  has  something  to  do  with  the  current  state  of  aviation 
safety.  Truth  be  told,  the  industry  is  extremely  safe,  and 
the  easy  fixes  have  been  identified  and  remedied.  What 
remains  to  be  addressed  is  the  small  fraction  of  accidents 
attributable  to  perhaps  the  most  complex  problem  facing 
aviation  today  -  human  error. 

A  closer  examination  of  the  current  aviation  accident 
record  has  revealed  that  anywhere  between  70-80%  of 
all  aviation  accidents  are  at  least  partially  attributable  to 
human  error  (Wiegmann  &  Shappell,  2003).  Therefore, 
it  stands  to  reason  that,  if  quantifiable  improvements 
in  aviation  safety  are  to  be  realized,  the  primary  focus 
should  be  on  the  human  operator  (i.e.,  aircrew)  and  those 
involved  with  the  safe  conduct  of  flight  (e.g.,  mechan¬ 
ics,  supervisors,  air  traffic  controllers)  rather  than  more 
traditional  areas  like  the  aircraft  itself. 

With  this  in  mind,  the  FAA  has  employed  the 
Human  Factors  Analysis  and  Classification  System 
(H FACS;  Wiegmann  &  Shappell,  2001, 2003;  Shappell 
&  Wiegmann,  2003,  2004)  to  identify  the  human  fac¬ 
tors  underlying  both  commercial  and  general  aviation. 
Principal  among  the  FAA’s  findings  using  HFACS  was 
the  observation  that,  while  previous  safety  programs  may 
have  impacted  other  areas  of  aviation,  there  has  been 
little  evidence  that  they  have  had  a  significant  impact  on 


any  specific  type  of  human  error  (Figure  1).  That  is  to 
say,  the  percentage  of  accidents  associated  with  aircrew 
error  (i.e.,  skill-based  errors,  decision  errors,  perceptual 
errors,  and  violations)  has  remained  relatively  stable 
since  1990. 

What  this  implies  is  that  intervention  strategies  imple¬ 
mented  in  the  1 990s  have  had,  at  best,  ubiquitous  effects 
on  the  errors  and  violations  committed  by  aircrew.  More 
likely,  however,  there  has  been  no  sustained  impact  of 
any  particular  intervention  program  (Shappell,  Detwiler, 
Holcomb,  Hackworth,  Boquet,  &  Wiegmann,  2006). 
The  latter  should  come  as  no  surprise,  given  that  prior 
to  these  FAA  studies,  no  comprehensive  analysis  of  air¬ 
crew  and  supervisory  error  had  been  conducted  using  a 
human  factors  approach  to  accident  causation.  Not  to 
mention  that  there  has  been  no  systematic  human  factors 
examination  of  the  current  or  proposed  safety  programs 
aimed  at  addressing  human  error. 

But  in  some  ways,  that  is  putting  the  proverbial  cart 
before  the  horse.  After  all,  while  HFACS  provided  a 
theoretically  derived  and  validated  framework  for  ac¬ 
cident/incident  investigation  and  analysis,  a  similar 
framework  did  not  exist  that  would  allow  the  FAA  and 
other  organizations  to  evaluate  the  potential  benefits  of 
current  and  proposed  human  error  intervention  strategies. 
So  the  better  question  may  be  whether  a  “human  factors” 
analysis  of  safety  programs  is  even  possible. 

NASA  Intervention  Strategies 

At  least  one  study  (Wiegmann  &  Rantanen,  2003) 
suggests  that  such  an  analysis  can  be  performed  using  a 
set  of  standards  derived  from  the  same  body  of  literature 
used  to  develop  HFACS.  In  their  book ,  A  Human  Factors 
Approach  to  Aviation  Accidents,  Wiegmann  and  Shappell 
(2003)  described  an  intervention  taxonomy  clustered 
around  four  broad  categories: 

1.  Environment  (e.g.,  the  control  of  temperature, 
noise,  vibration,  lighting) 

2.  Human  (personnel  selection,  incentives,  training, 
teamwork,  communication,  etc.) 

3.  Machine  (engineering  design,  capacity,  etc.) 

4.  Task  (ordering/ timing  of  events,  procedures,  stan¬ 
dardization,  etc.) 
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Figure  1.  Percentage  of  unsafe  acts  committed  by 
aircrew  during  general  aviation  (Panel  A)  commut¬ 
er  (Panel  B)  and  air  carrier  (Panel  C)  operations 
by  year.  Note  that  percentages  will  not  add  up  to 
100%  due  to  multiple  causal  factors  associated 
with  most  accidents. 
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Figure  2.  Percentage  of  NASA  safety  programs 
within  each  intervention  category. 


Using  this  framework,  Wiegmann  and  Rantanen 
(2003)  examined  a  variety  of  technologies  developed  by 
NASA’s  aviation  safety  program  (AvSP).  From  energy 
absorbing  seats,  restraints,  and  structures  to  synthetic 
vision,  each  safety  program  was  classified  within  one 
of  the  four  intervention  categories.  As  shown  in  Figure 
2,  they  concluded  that  NASA’s  primary  intervention 
strategies  targeted  the  machine  rather  than  the  human, 
environment,  or  task.  Two  programs,  Incident  Report¬ 
ing  Enhancement  Tools  and  Fast-time  Simulation  of 
System-wide  Risks,  were  considered  unclassifiable  by 
the  raters  using  these  categories. 

In  a  separate  part  of  their  study,  Wiegmann  &  Ran¬ 
tanen  (2003)  examined  the  NASA  technologies  using 
the  FIFACS  framework.  Surprisingly,  it  was  determined 
that  nearly  half  of  the  technologies  that  NASA  was 
developing  were  rated  as  having  no  impact  on  aircrew 
error.  What’s  more,  those  that  might  have  an  impact 
primarily  targeted  decision  errors,  by  providing  better 
information,  automation,  and  training.  An  even  smaller 
percentage  of  the  technologies  targeted  aircrew  error,  in 
general,  and  only  one  of  the  products  primarily  targeted 
skill-based  errors  -  the  most  frequent  human  error  fac¬ 
ing  both  commercial  and  general  aviation.  None  of  the 
products  primarily  targeted  violations,  another  area  of 
concern  within  civilian  aviation  operations. 

Purpose 

Clearly,  if  improvements  in  safety  are  to  be  real¬ 
ized,  a  more  systematic  methodology  is  needed  for 
generating  intervention/prevention  strategies  that  can 
tie  into  human  error  frameworks  like  IT  FACS.  Such  a 
methodology  would  help  ensure  that  factors  affecting 
human  performance  are  addressed  at  multiple  levels 
and  from  multiple  directions,  thereby  facilitating  the 
development  of  effective  intervention  strategies  rather 
than  a  single,  narrowly  focused  design  fix. 

This  report  describes  two  studies  that  build  upon 
the  methodology  originally  described  by  Wiegmann 
and  Shappell  (2003)  and  used  by  Wiegmann  and 
Rantanen  (2003)  with  NASA  safety  programs.  The 
first  study  describes  an  independent  validation  of  the 
four  intervention  methodologies  using  safety  recom¬ 
mendations  from  the  NTSB.  The  second  describes  the 
examination  of  proposed  FAA  aviation  safety  programs 
using  a  prototype  intervention  matrix  that  maps  the 
unsafe  acts  of  operators  (i.e.,  skill-based  errors,  decision 
errors,  perceptual  errors,  and  violations)  onto  several 
intervention  approaches. 


2 


STUDY  1:  ANALYSIS  OF  NTSB 
RECOMMENDATIONS 

Investigating  accidents,  identifyingpotential  interven¬ 
tions,  and  issuing  safety  recommendations  are  central  to 
any  safety  program  and  as  such  are  a  major  function  of 
local,  state,  and  federal  safety  boards.  Indeed,  one  such 
national  entity,  the  NTSB,  cites  safety  recommendations 
as  "...  the  most  important  part  of  [their]  mandate...” 
(NTSB,  2002). 

Ideally,  safety  recommendations,  when  adopted  by 
cognizant  organizations,  will  positively  influence  future 
operations  in  the  field  and  thereby  improve  overall  sys¬ 
tem  safety.  However,  recommendations  are  just  that  ... 
recommendations  and,  as  such,  are  not  always  adopted. 
Moreover,  they  are  often  based  solely  on  isolated  events 
or  at  best  a  few  events  over  a  very  short  period  of  time 
rather  than  more  global  analyses  of  the  system  as  a  whole. 
While  these  interventions  may  solve  a  local  or  single-point 
problem,  they  often  do  not  have  far-reaching  impact. 

Further  complicating  matters,  many  domains  such 
as  aviation  and  their  corresponding  safety  boards  have 
traditionally  strong  relations  with  quantitative  disciplines 
like  engineering  and  physics.  Consequently,  while  these 
organizations  may  be  especially  adept  at  dealing  with 
mechanical  issues,  they  tend  to  be  less  robust  when 
dealing  with  organizational  or  human-centered  aspects 
of  accidents  like  human  error,  organizational  failure, 
communication,  and  risk  assessment  (Stoop,  2002). 

Recognizing  this,  the  NTSB,  like  many  safety  entities, 
has  integrated  human  factors  experts  into  their  organiza¬ 
tion,  presumably  leading  to  recommendations  that  ad¬ 
dress  the  entire  system  rather  than  a  single  engineering  or 
mechanical  aspect,  per  se.  However,  employing  human 
factors  experts  alone  does  not  necessarily  translate  into 
a  breadth  of  interventions.  The  question  remains,  what 
specific  intervention  approaches  does  the  NTSB  employ? 
In  other  words,  does  the  NTSB  tend  to  be  uni-dimen- 
sional  (like  NASA)  or  multi-dimensional  with  regard  to 
specific  intervention  approaches? 

METHOD 

NTSB  Safety  Recommendations 

To  examine  this  question,  aviation  safety  recommen¬ 
dations  associated  with  commercial  (14  CFR  Part  121 
-  air  carrier  and  Part  1 35  -  commuter)  aviation  accidents 
occurring  between  1998  and  2004  were  obtained  from 
the  NTSB’s  official  Website  (www.ntsb.gov).  Of  the 
147  commercial  aviation  accidents  reports  that  were 
completed  at  the  time  of  this  study,  622  unique  safety 
recommendations  were  identified.  However,  several  of  the 
recommendations  consisted  of  compound  solutions.  In 


those  cases,  the  original  recommendation  was  separated 
into  sub-recommendations  while  preserving  the  intent  of 
the  NTSB.  This  resulted  in  a  revised  list  of  872  unique 
recommendations  for  further  analysis. 

Clustering  Process 

The  recommendations  were  independently  clustered 
into  categories  by  two  analysts  (one  with  a  doctoral-level 
background  in  psychology,  the  other  with  a  graduate 
background  in  engineering)  based  on  their  similarities. 
The  analysts  were  not  instructed  to  use  any  predefined 
taxonomy  or  classification  scheme.  They  were  simply 
instructed  to  independently  assign  each  recommendation 
to  categories  of  their  choosing,  based  upon  the  nature  of 
the  recommendation. 

Not  surprising,  given  the  vagueness  of  the  instructions, 
there  were  some  differences  in  the  terms  used  by  the  two 
analysts,  but  there  were  also  strong  similarities.  Wherever 
disagreements  occurred,  the  analysts  were  asked  to  discuss 
their  clustering  heuristic  and  to  agree  on  a  single  clas¬ 
sification  scheme.  In  the  end,  all  872  recommendations 
were  classified  based  on  their  underlying  similarities  by 
two  independent  analysts,  who  later  came  to  a  consensus 
on  the  number  and  labels  for  each  of  these  clusters. 

Results 

Ultimately,  the  analysts  generated  nine  unique  cat¬ 
egories  of  recommendations,  which  included  the  design 
of  parts/displays,  procedures,  communication,  training, 
requests  to  conduct  focused  studies,  rules,  manuals, 
inspection,  and  human  resources.  These  nine  categories 
were  then  further  clustered  into  four  larger  categories 
based  on  their  similarities:  1)  administrative/organiza- 
tional;  2)  mechanical/  engineering;  3)  human/crew;  and 
4)  task/mission.  Each  category  and  their  accompanying 
subcategories  are  briefly  described  in  Table  1 . 

Distribution  of  recommendations 

On  average,  roughly  six  recommendations  spread 
across  just  under  three  (2.8)  intervention  subcategories 
were  observed  per  accident.  The  actual  distribution  of 
recommendations  across  the  intervention  categories  and 
subcategories  is  presented  in  Table  2. 

From  a  global  perspective,  it  appears  that  roughly 
two-thirds  of  the  recommendations  were  either  admin¬ 
istrative/organizational  or  mechanical/  engineering  fixes. 
However,  nearly  a  quarter  of  the  recommendations  were 
aimed  at  either  the  task  or  mission. 

Surprisingly  few  interventions  directly  targeted  opera¬ 
tors  (aircrew),  even  though  previous  studies  repeatedly 
show  that  more  major  accidents  have  been  attributed  to 
human  error  than  to  any  other  single  cause  (Wiegmann 
&  Shappell,  2003;  Boquet  et  al. ,  in  review;  Detwiler  et  al. , 
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Table  1.  Proposed  categories  and  sub¬ 
categories  of  NTSB  recommendations. 

Administrative/Organizational 

Rules/Regulations/Policies :  Issuing,  modifying, 
establishing,  amending,  and/or  reviewing 
policies,  rules,  or  regulations. 

Information  Management/  Communication : 
Improvements  in  disseminating,  storing, 
archiving  and  publishing  information.  Also 
included  are  recommendations  regarding 
collecting  data;  issuing  information  bulletins, 
advisory  circulars,  and  reporting  activity. 
Research/  Special  Study.  Conducting  research  to 
determine  the  impact  of  recent  technological 
advances  or  call  for  special  studies  to  review 
processes,  develop/validate  methodologies, 
evaluate  the  feasibility  of  safety  equipment, 
and/or  conduct  surveys. 

Human  Resource  Management.  Adequacy  of 
staff  in  specific  situations,  the  need  for  additional 
personnel,  and  the  evaluation  of  individual  skills 
of  employees. 

Mechanical/Engineering 

Design/Repair.  Specific  manufacturing  changes 
including  the  design  of  parts.  Also  included  is 
the  modification,  replacement,  removal  and/or 
installation,  or  repair  of  parts  and  equipment. 
Inspection :  Maintenance  inspections, 
overhauling,  detecting  damage,  including  day-to- 
day  operations  such  as  inspecting  fuel,  oil  level, 
and  recommended  safety  checks. 

Human  /  Crew 

Training:  Reviewing,  developing,  and 
implementing  training  programs.  Also  included 
is  the  training  of  personnel  in  handling 
emergencies. 

Task/Mission 

Procedures'.  Amending,  reviewing,  modifying, 
revising,  establishing,  developing,  and  validating 
procedures. 

Manuals'.  Reviewing,  revising,  issuing, 
amending,  and  modifying  manuals,  bulletins, 
checklists,  and  other  instructions  or  guidance. 


2006;  Shappell  et  al.,  in  press).  It  has  also  been  observed 
that  wider  systemic  issues,  including  the  managerial 
and  regulatory  context  of  aviation  operations,  were  also 
mentioned  in  a  large  number  of  reports  (Holloway  & 
Johnson,  2004;  Johnson,  in  review),  even  though  this 
does  not  seem  to  be  reflected  in  the  accident  record. 

A  closer  examination  revealed  that,  similar  to  Wieg- 
mann  and  Rantanen’s  study  of  NASA  safety  programs, 
design  fixes  constituted  the  largest  percentage  of  any 
individual  type  of  recommendation  made  by  the  NTSB 


Table  2.  Percentage  of  recommendations 
associated  with  each  intervention  category. 


Intervention  Category 

% 

11 

Administrative/Organizational 

34.18 

Rules/Regulations/Policies 

9.29 

81 

Information  management/  Comm. 

13.76 

120 

Research/Special  study 

10.44 

91 

Human  resource  management 

0.69 

6 

Mechanical/Engineering 

31.20 

Design/Repair 

23.17 

202 

Inspection 

8.03 

70 

Human/Crew 

11.47 

Training 

11.47 

100 

Task/Mission 

23.16 

Procedures 

14.56 

127 

Manuals 

8.60 

75 

(23.17%)  -  nearly  twice  as  many  as  any  other  category. 
Considerably  fewer  were  aimed  at  procedures,  training, 
information  management/  communication,  and  the 
other  subcategories. 

SUMMARY 

To  date,  there  have  been  few  attempts  to  systemati¬ 
cally  study  recommendations  generated  by  investigative 
organizations  like  the  NTSB.  This  is  unfortunate,  because 
the  results  of  such  studies  may  help  in  understanding 
why  accident  rates  have  stabilized  over  the  last  several 
decades  and  could  lead  to  the  development  of  more  ef¬ 
fective  intervention  strategies.  For  example,  in  this  study 
alone  there  were  four  broad  categories  of  interventions 
identified,  comprising  nine  unique  categories  of  recom¬ 
mendations. 

When  examining  the  breadth  and  scope  of  NTSB  rec¬ 
ommendations,  even  at  this  level,  it  appears  that  current 
safety  recommendations  in  aviation  tend  to  focus  more 
on  improving  the  design  of  systems  or  some  manner  of 
organizational  change  rather  than  focusing  on  operational 
personnel.  While  these  recommendations  are  obviously 
well-intentioned  and  often  specific  to  a  particular  accident, 
they  may  be  misplaced  or  too  narrow  in  scope.  This  may 
help  explain  why  the  percentage  of  accidents  associated 
with  human  error  has  not  changed  over  the  last  1 5  years 
(Wiegmann  &  Shappell,  2003;  Wiegmann  et  al.,  2005; 
Shappell  et  al.,  in  press). 

This  is  not  to  say  that  the  design  of  new  technology 
will  not  have  a  significant  impact  on  how  people  perform. 
After  all,  advances  in  aviation  technology  and  engineering 
have  accounted  for  marked  reductions  in  the  aviation 
accident  rate  since  the  late  1950s.  On  the  other  hand, 
some  of  these  advances  have  led  to  new,  occasionally 
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catastrophic,  errors  (e.g.,  autopilot-mode  errors,  Sarter 
&  Woods,  1992;  1994).  What  appears  to  be  required  is 
a  broader,  systematic  approach  to  accident  intervention, 
particularly  if  we  are  to  effectively  address  human  error 
within  aviation  operations.  But  how  can  this  be  done? 

To  ensure  that  safety  professionals  generate  effective 
intervention  strategies,  rather  than  a  single  “knee  jerk” 
fix  to  a  problem,  knowledge  of  all  viable  interventions  is 
required.  Towards  these  ends,  the  present  study  suggests 
that  there  are  at  least  four  broad  categories  of  interven¬ 
tions  that  appear  tenable  within  the  aviation  industry. 
These  are  Administrative/Organizational,  Human/Crew, 
Mechanical/Engineering,  and  Task/Procedure. 

These  four  approaches  differed  slightly  from  those 
previously  proposed  by  Wiegmann  and  Shappell  (2003) 
and  utilized  by  Wiegmann  and  Rantanen  (2003)  to  ana¬ 
lyze  NASA  safety  programs.  One  category  that  naturally 
surfaced  from  the  present  analysis,  but  was  missing  from 
the  Wiegmann  and  Rantanen  study,  was  Administra¬ 
tive/Organizational  interventions.  In  contrast,  “envi¬ 
ronmental”  interventions  did  not  appear  in  the  current 
study  but  were  present  in  the  NASA  study  (Wiegmann 
&  Rantanen,  2003). 

In  the  end,  the  question  is  not  whether  or  not  there 
are  three,  four,  five,  or  more  approaches  to  identifying 
potential  accident  interventions  as  much  as  there  is  de¬ 
finitively  more  than  one.  Exactly  what  those  approaches 
are  remains  to  be  fully  explored.  However,  the  five  ap¬ 
proaches  identified  between  the  present  study  and  the 
investigation  conducted  by  Wiegmann  and  Rantanen 
(2003)  is  a  reasonable  first  start. 

STUDY  2:  HFIX  ANALYSIS  OF  JSAT/ 
JSIT  RECOMMENDATIONS 

Identifying  viable  approaches  for  intervening,  however, 
is  only  the  first  step.  The  ability  to  map  interventions  onto 
specific  types  of  human  error  is  also  vitally  important. 
In  other  words,  simply  generating  a  variety  of  interven¬ 
tions  across  several  domains,  whether  they  are  human, 
mechanical,  environmental,  and  so  on,  is  likely  to  be 
ineffective  unless  such  interventions  directly  target  the 
problem  area. 

Given  that  human  error  continues  to  be  the  largest 
contributor  to  commercial  and  general  aviation  accidents, 
it  makes  sense  to  map  different  interventions  against 
specific  error  forms.  What  is  needed  is  a  theoretical  frame¬ 
work  that  captures  the  underlying  causal  mechanisms 
of  human  error  along  with  the  intervention  approaches 
identified  in  Study  1 . 


Human  Factors  Analysis  and  Classification  System 

Such  an  error  framework  already  exists  and  is  widely 
used  within  the  aviation  industry.  This  framework, 
the  Human  Factors  Analysis  and  Classification  System 
(HFACS),  describes  two  general  categories  of  unsafe 
acts  that  operators  commit:  errors  -  the  honest  mistakes 
individuals  make  every  day,  and  violations  -  the  willful 
disregard  for  the  rules  and  regulations  of  safety.1  Within 
those  two  overarching  categories,  HFACS  describes  three 
types  of  errors  (decision,  skill-based,  and  perceptual)  and 
two  types  of  violations  (routine  and  exceptional).  Each 
is  briefly  described  below. 

Errors 

One  of  the  more  common  error  forms,  decision  er¬ 
rors,  represents  conscious,  goal-intended  behavior  that 
proceeds  as  designed,  yet  the  plan  proves  inadequate  or 
inappropriate  for  the  situation.  Often  referred  to  as  “honest 
mistakes,”  these  unsafe  acts  typically  manifest  as  poorly 
executed  procedures,  improper  choices,  or  simply  the 
misinterpretation  or  misuse  of  relevant  information. 

In  contrast  to  decision  errors,  the  second  error  form, 
skill-based  errors,  occurs  with  little  or  no  conscious  thought. 
Just  as  little  thought  goes  into  turning  one’s  steering  wheel 
or  shifting  gears  in  an  automobile,  basic  flight  skills  such 
as  stick  and  rudder  movements  and  visual  scanning  often 
occur  without  thinking.  The  difficulty  with  these  highly 
practiced  and  seemingly  automatic  behaviors  is  that  they 
are  particularly  susceptible  to  attention  and/or  memory 
failures.  As  a  result,  skill-based  errors  such  as  the  break¬ 
down  in  visual  scan  patterns,  inadvertent  activation/de- 
activation  of  switches,  forgotten  intentions,  and  omitted 
items  in  checklists  often  appear.  Even  the  manner  in  (or 
skill)  which  one  flies  an  aircraft  (aggressive,  tentative,  or 
controlled)  can  affect  safety. 

While  decision  and  skill-based  errors  have  domi¬ 
nated  most  accident  databases  and  have,  therefore,  been 
included  in  most  error  frameworks,  the  third  and  final 
error  form,  perceptual  errors,  has  received  comparatively 
less  attention.  No  less  important,  perceptual  errors  occur 
when  sensory  input  is  degraded,  or  “unusual,”  as  is  often 
the  case  when  flying  at  night,  in  the  weather,  or  in  other 
visually  impoverished  environments.  Faced  with  acting 
on  imperfect  or  incomplete  information,  aircrews  run 
the  risk  of  misjudging  distances,  altitude,  and  decent 
rates,  as  well  as  responding  incorrectly  to  a  variety  of 
visual/vestibular  illusions. 


1  A  complete  description  of  the  entire  HFACS  framework,  including 
all  4  tiers  and  1 9  causal  categories,  can  be  found  in  Wiegmann  and 
Shappell,  2003. 
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Violations 

Although  there  are  many  ways  to  distinguish  among 
types  of  violations,  two  distinct  types  have  been  identi¬ 
fied  based  on  their  etiology.  Routine  violations  tend  to  be 
habitual  by  nature  and  are  often  enabled  by  a  system  of 
supervision  and  management  that  tolerates  such  depar¬ 
tures  from  the  rules  (Reason,  1990).  Often  referred  to 
as  “bending  the  rules,”  the  classic  example  is  that  of  the 
individual  who  drives  his/her  automobile  consistently  5- 
1 0  mph  faster  than  allowed  by  law.  While  clearly  against 
the  law,  the  behavior  is,  in  effect,  sanctioned  by  local 
authorities  (police)  who  often  will  not  enforce  the  law 
until  speeds  in  excess  of  10  mph  over  the  posted  limit 
are  observed. 

Exceptional  violations,  on  the  other  hand,  are  isolated 
departures  from  authority,  neither  typical  of  the  individual 
nor  condoned  by  management.  For  example,  while  driving 
65  in  a  55  mph  zone  might  be  condoned  by  authorities, 
driving  105  mph  in  a  55  mph  zone  certainly  would  not. 
It  is  important  to  note  that  while  most  exceptional  viola¬ 
tions  are  appalling,  they  are  not  considered  “exceptional” 
because  of  their  extreme  nature.  Rather,  they  are  regarded 
as  exceptional  because  they  are  neither  typical  of  the 
individual  nor  condoned  by  authority. 

Human  Factors  Intervention  Matrix  (HFIX) 

A  prototype  matrix,  called  the  Human  Factors  Inter¬ 
vention  Matrix  (HFIX),  pits  the  unsafe  acts  individuals 
commit  against  the  five  different  intervention  approaches 
presented  in  Figure  3.  The  utility  of  such  a  framework 
seems  intuitive.  For  example,  if  one  were  interested  in 
developing  interventions  to  address  decision  errors,  the 
goal  would  be  to  identify  prospective  interventions  within 
each  approach  (i.e.,  organizational/administrative,  hu¬ 
man/crew,  etc.),  thereby  ensuring  that  the  widest  array 
of  interventions  were  considered.  By  mapping  prospec- 
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tive  interventions  onto  the  matrix,  it  would  be  readily 
apparent  if  the  scope  of  a  proposed  program  was  uni-  or 
multi-dimensional. 

Alternatively,  a  framework  like  HFIX  could  be  used 
proactively  to  determine  which  areas  an  organization  has 
“covered”  and  where  gaps  exist  in  the  current  safety  pro¬ 
gram  given  current  trends  in  the  error  data.  For  instance, 
if  you  knew  that  the  largest  threat  to  safety  within  your 
organization  was  skill-based  errors,  followed  by  decision 
errors,  violations,  and  perceptual  errors  (as  is  the  case 
with  general  and  commercial  aviation  in  the  U.S.),  HFIX 
could  be  used  to  determine  if  your  proposed  and  future 
interventions  have  the  potential  to  address  those  needs 
and  which  areas  are  currently  being  targeted. 

Hence,  the  purpose  of  Study  2  was  to  determine  if 
such  an  approach  could  be  used  within  the  FAA  and 
which  types  of  human  error  might  be  affected  by  current 
and  future  interventions.  In  a  sense,  this  analysis  would 
provide  a  “benchmark”  of  current  FAA  intervention  ef¬ 
forts.  When  combined  with  existing  HFACS  data  (e.g., 
Shappell  et  ah,  in  press;  Detwiler  et  ah,  2006;  Wiegmann 
et  ah,  2005)  possible  gaps,  if  any,  may  be  identified. 

FAA  Safer  Skies  Initiative 

As  part  of  the  FAA’s  Safer  Skies  initiative,  three  teams 
of  experts  from  government,  employee  advocacy  groups 
(e.g.,  the  National  Air  Traffic  Controllers  Association), 
the  aviation  industry,  and  academia  were  formed  to  ad¬ 
dress  civilian  aviation  accidents.  Two  of  those  teams,  the 
Commercial  Aviation  Safety  Team  (CAST)  and  General 
Aviation  Joint  Steering  Committee  (GAJSC) ,  were  formed 
to  address  specific  threats  to  commercial  and  general 
aviation,  respectively. 

With  the  CAST  and  the  GAJSC  providing  oversight, 
three  working  groups  were  formed:  1 )  Joint  Safety  Analysis 
Teams  (JSATs),  2)  Joint  Safety  Implementation  Teams 
(JSITs),  and  3)  Joint  Implementation  Monitoring  Teams 
(JIMTs).  Particularly  germane  to  this  study  were  outcomes 
derived  from  the  JSAT  and  JSIT  working  groups  since 
they  represented  current  and  future  interventions  neces¬ 
sary  to  address  human  error  associated  with  commercial 
and  general  aviation  accidents.  In  particular,  this  study 
was  interested  in  the  recommendations  from  JSAT/JSIT 
teams  examining  accidents  associated  with: 

•  Controlled  flight  into  terrain 

•  Approach  and  landing 

•  Loss  of  control 

•  Runway  incursions 

•  Weather 

•  Pilot  decision-making 


Figure  3.  The  “Human  Factors  intervention 
matrix”  (HFIX). 
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METHOD 


RESULTS 


JSAT  and  JSIT  Recommendations 

JSAT  and  JSIT  reports  were  collected  from  each 
CAST  and  GA  JSC  committee  by  researchers  at  the 
Civil  Aerospace  Medical  Institute.  After  eliminating 
duplicate  recommendations,  a  comprehensive  list  was 
compiled  electronically  for  classification.  The  final  list 
of  614  unique  recommendations  was  then  randomized 
to  reduce  bias. 

Categorization  of  the  Data 

Eighteen  Master  of  Aeronautical  Science  candidates 
were  recruited  from  Embry- Riddle  Aeronautical  Univer¬ 
sity  for  Study  2.  Each  had  experience  in  the  aviation  com¬ 
munity  as  either  a  pilot,  maintainer,  or  at  an  administrative 
level,  and  all  had  successfully  completed  a  minimum  of 
one  graduate-level  human  factors  course. 

After  a  roughly  4-hour  training  session  on  the  EIFACS 
and  EIFIX frameworks,  subjects  were  randomly  assigned 
to  one  of  six  groups.  Each  3-person  team  was  then  ran¬ 
domly  assigned  roughly  one-sixth  of  the  recommenda¬ 
tions  to  classify. 

Each  team  member  was  instructed  to  independently 
classify  each  recommendation  into  only  one  of  the  five 
intervention  approaches  (i.e.,  organizational/  adminis¬ 
trative,  human/crew,  mechanical/engineering,  task/mis- 
sion,  or  physical  environment).  In  addition,  they  were 
instructed  to  identify  any  EIFACS  Unsafe  Acts  categories 
they  felt  the  intervention  would  impact. 

After  the  initial  rating,  team  members  were  permit¬ 
ted  to  discuss  their  classification  within  their  group  to 
resolve  any  differences.  A  final,  consensus  classification 
for  each  recommendation  was  then  provided  for  further 
analysis. 
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Figure  4.  Percentage  of  JSAT/JSIT  recom¬ 
mendations  classified  by  intervention  approach 
and  specific  HFACS  Unsafe  Act  addressed. 


The  results  of  both  classification  tasks  are  presented 
in  Figure  4.  Several  observations  can  be  made  from  the 
data.  First,  as  with  the  NTSB  recommendations,  a  large 
percentage  (36.6%)  of  the  JSAT/JSIT  recommendations 
were  directed  at  organizational/ administrative  levels.  Like¬ 
wise,  several  (22.2%)  of  the  recommendations  involved 
technological/engineering  approaches.  However,  unlike 
the  NTSB  where  relatively  few  recommendations  targeted 
the  human,  nearly  one-third  of  those  obtained  from  the 
J SAT /J SITs  did  so.  This  may  be  because,  unlike  the  NTSB 
recommendations,  we  selectively  chose  those  JSAT/JSIT 
reports  that  addressed  human  error  issues  like  pilot  deci¬ 
sion-making  and  runway  incursions.  In  that  sense,  the 
JSAT/JSIT  data  were  much  more  homogenous  (i.e.,  they 
did  not  contain  non-human  related  accidents)  and  the  pro¬ 
posed  interventions  may  simply  reflect  that  inherent  bias. 
However,  if  that  were  true,  one  might  actually  expect  that 
an  even  larger  percentage  of  the  recommendations  would 
target  the  human/ crew  than  was  actually  observed. 

When  examining  the  HFACS  classifications,  remember 
that,  unlike  the  specific  approaches  to  accident  interven¬ 
tions  where  subjects  were  instructed  to  select  only  one 
approach,  they  were  permitted  to  select  all  of  the  HFACS 
Unsafe  Act  categories  that  they  felt  would  be  impacted  by 
a  given  recommendation.  Therefore,  unlike  the  interven¬ 
tion  approaches  whose  percentages  added  up  to  1 00%, 
the  total  percentages  associated  with  each  Unsafe  Act 
category  did  not. 

Perhaps  not  unexpected,  interventions  aimed  at  deci¬ 
sion  errors  were  associated  with  nearly  three  out  of  every 
four  JSAT/JSIT  recommendations  examined.  In  contrast, 
skill-based  errors  were  associated  with  roughly  50%  of 
the  recommendations  followed  by  perceptual  errors 
(37.6%)  and  violations  (26.9%).  Ofnote,  these  numbers 
are  slightly  different  than  the  percentage  of  accidents  as¬ 
sociated  with  each  type  of  error  where  skill-based  errors 
account  for  between  45-80%  of  the  accidents,  depending 
on  whether  one  is  talking  about  commercial  or  general 
aviation,  respectively  (see  Figure  1 ) .  Likewise,  roughly  1/3 
of  the  accidents  were  associated  with  decision  errors,  yet 
72.6%  of  the  interventions  have  some  component  that 
will  potentially  affect  pilot  decision-making. 

This  is  not  to  say  that  there  should  be  a  one-to-one 
relationship  between  the  percentage  of  accidents  as¬ 
sociated  with  a  given  error  category  and  the  percentage 
of  recommendations  aimed  at  addressing  these  errors. 
After  all,  it  may  take  more  effort  to  address  one  error 
form  than  another,  or  more  interventions  may  naturally 
address  pilot  decision-making.  In  either  case,  the  global 
analysis  presented  here  suggests  that  additional  review  of 
this  apparent  incongruity  is  necessary. 
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Perhaps  more  important,  however,  was  the  mapping  of 
each  intervention  within  both  the  intervention  approach 
and  the  HFACS  Unsafe  Acts  category  (Figure  4).  As  can 
be  seen  (white  boxes) ,  three  of  the  20  possible  boxes  (orga¬ 
nizational/  administrative  by  decision  error,  human/crew 
by  decision  error,  and  human/crew  by  skill-based  error) 
contained  20%  or  more  of  the  JSAT/JSIT  interventions. 
On  the  surface,  this  appears  to  reflect  a  narrow  rather 
that  a  broad  approach  to  accident  intervention/mitiga- 
tion  by  these  committees.  It  is  not  that  the  interventions 
contained  within  these  categories  will  not  be  effective, 
just  that  other,  potentially  equally  viable,  interventions 
may  have  been  overlooked. 

It  is  interesting  to  note,  however,  that  if  one  examines 
those  boxes  that  contained  between  1 0-20%  of  the  possible 
interventions,  nearly  all  of  the  remaining  boxes  among 
the  organizational/  administrative,  human/crew,  and 
technology/  engineering  approaches  are  included.  What 
was  not  accounted  for  were  human/crew  and  technol¬ 
ogy/engineering  approaches  dealing  with  violations  ofthe 
rules  and  regulations.  Obviously,  these  approaches  might 
prove  beneficial  if  an  organization  wanted  to  modify  or 
curtail  a  particular  unsafe  pattern  of  behavior  (e.g.,  flight 
into  instrument  conditions  while  on  a  visual  flight  rules 
flight  plan)  through  training  or  technological  means. 

More  notable  was  the  general  lack  of  interventions 
targeting  the  specific  task/mission  of  the  aircrews  or  the 
environment  they  are  faced  with.  Perhaps  a  closer  examina¬ 
tion  of  the  operations  these  aircrews  are  engaged  in  or  the 
environments  they  are  expected  to  operate  in  is  warranted. 
In  any  event,  there  may  have  been  options  along  these  lines 
that  were  not  considered  by  these  select  committees. 

SUMMARY 

Ideally,  tools  such  as  FIFIX  provide  a  Gestalt  of  the 
safety  program  as  a  whole  rather  than  an  item-by-item 
accounting  of  each  intervention  in  an  organization.  After 
all,  it  is  hard  to  know  if  pieces  are  missing  in  a  puzzle  until 
you  put  them  together.  HFIX  allows  administrators  and 
safety  managers  to  put  the  intervention  pieces  together 
in  such  a  way  that  they  can  get  a  “quick  look”  at  the 
strengths  and  weaknesses  of  their  programs .  Additionally, 
it  provides  decision  makers  within  an  organization  the 
ability  to  ensure  that  a  broad  spectrum  of  interventions 
has  been  considered.  After  all,  only  the  most  elementary 
of  puzzles  is  comprised  of  just  a  few  pieces;  obviously, 
something  as  complex  as  human  error  in  aviation  will 
consist  of  a  number  of  pieces. 

That  being  said,  the  results  from  Study  2  using 
JSAT/JSIT  interventions,  although  clearly  more  multi¬ 
dimensional  than  NASA’s  safety  programs,  still  did  not 
appear  to  fully  address  the  current  accident  trends  in 


commercial  and  general  aviation.  At  least  on  the  surface, 
it  appears  that  there  are  gaps  in  the  safety  program  that 
should  be  addressed. 

For  example,  there  was  an  apparent  bias  toward  inter¬ 
ventions  aimed  at  pilot  decision-making,  particularly  those 
utilizing  organizational  and  human  approaches.  While  this 
is  not  inherently  bad,  previous  HFACS  analyses  suggest 
that  additional  effort  should  be  placed  on  skill-based  errors 
and  violations,  two  areas  that  appear  underrepresented, 
given  current  trends  in  the  accident  data. 

Also  noteworthy,  few  interventions  attempted  to 
modify/ change  the  task  itself  or  the  environment.  A 
closer  examination  of  the  actual  types  of  errors  may 
suggest  changes  in  routes  people  fly  or  the  actual  type  of 
flights  being  flown. 

However,  while  HFIX  may  prove  useful  when  generat¬ 
ing  comprehensive  intervention  strategies,  organizations 
simply  cannot  implement  every  recommendation.  Other 
factors  may  need  to  be  considered  before  employing 
a  given  intervention.  Factors  such  as  effectiveness  (i.e., 
what  is  the  likelihood  that  it  will  work?),  cost  (i.e.,  can 
the  organization  afford  the  intervention?),  feasibility 
(i.e.,  how  easy  will  the  intervention  be  to  implement 
or  does  it  actually  exist?),  and  acceptability  (i.e.,  will  the 
workforce  accept  the  proposed  intervention?)  all  must 
be  considered. 

As  such,  HFIXmay  actually  be  HFIX3  mapping  human 
error  against  the  intervention  approaches  and  evaluations 
criteria  (Figure  5).  Although  it  may  appear  complex,  in 
reality  organizational  decision  makers  utilize  this  third 
dimension  all  the  time.  To  apply  it  to  the  two-dimen¬ 
sional  HFIX  framework  is  really  not  that  great  a  leap. 
However,  even  without  this  third  dimension,  the  mapping 
of  specific  interventions  onto  a  matrix  that  combines  the 
five  intervention  approaches  with  general  categories  of 
human  error  can  provide  a  broader  perspective  of  the 
FAA’s  safety  programs. 


Figure  5.  The  HFIX3  framework. 
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Historically,  most  safety  professionals  have  been 
heavily  acculturated  by  their  own  academic  disciplines. 
While  such  indoctrination  and  training  can  facilitate  the 
development  of  highly  specialized  interventions,  it  can 
also  lead  to  “mitigation  myopia,”  in  which  provincial 
prevention  measures  prevail.  This  is  not  to  say  that  such 
dogmatism  is  intentional.  Rather,  these  biases  or  “cognitive 
constraints”  placed  on  our  creativity  are  quite  simply  the 
natural  byproducts  of  the  acculturation  process  associ¬ 
ated  with  each  academic  discipline  or  society  in  which 
one  lives.  It  should  come  as  no  surprise  then  that,  while 
engineers  have  traditionally  blamed  the  operator  for  er¬ 
rors  and  behaviorists  have  wanted  to  fault  system  design 
for  inducing  errors,  the  fixes  have  been  predictable.  That 
is,  engineers  tend  to  recommend  engineering  solutions, 
and  psychologists  tend  to  recommend  behavioral/hu¬ 
man-centered  fixes.  In  a  broader  sense,  even  societies 
that  emphasize  individual  responsibility  for  one’s  own 
actions  tend  to  emphasize  punitive  fixes. 

In  essence,  safety  recommendations  are  not  simply 
based  on  empirical  findings  surrounding  an  accident. 
Rather,  they  are  based  on  one’s  philosophical  view  of 
what  actually  constitutes  a  “cause”  of  an  event,  coupled 
with  one’s  own  biased  view  of  how  changes  in  human  or 
system  behavior  can  even  be  accomplished.  Therefore, 
thinking  “outside  the  box”  when  it  comes  to  generating 
intervention  strategies  is  extremely  difficult  to  do;  yet 
failure  to  do  so  can  leave  other  potentially  viable  and 
effective  alternatives  unexplored. 

What  shall  we  say  then,  that  we  are  forever  helpless 
victims  of  our  own  acculturation  and  training?  Absolutely 
not!  Just  like  other  cognitive  biases  (e.g.,  confirmation 
bias  and  hindsight  bias),  we  must  first  acknowledge  and 
recognize  the  potential  impact  that  our  own  “mitiga¬ 
tion  bias”  has  on  constraining  our  judgment  and  then 
generate  tools  and  techniques  for  circumventing  these 
constraints. 

In  the  end,  perhaps  Reason  (2005)  put  it  best  when 
he  said,  “[Human  errors]  are  like  mosquitoes.  They  can 
be  swatted  one  by  one,  but  they  still  keep  coming.  The 
best  remedies  are  to  create  more  effective  defenses  and 
to  drain  the  swamps  in  which  they  breed.”  Where  the 
HFACS  framework  provides  a  view  of  the  swamp,  HFIX 
makes  certain  that  we  are  draining  the  right  swamp  in 
the  most  efficient  and  thorough  manner. 
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